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Background / Context: 

Description of prior research and its intellectual context. 

A widely cited meta-analysis of the effects of comprehensive school reform and student 
achievement (Borman, Hewes, Overman, and Brown, 2003) found Modern Red Schoolhouse- 
developed whole-school intervention programs to boost student achievement outcomes at rates 
on par with or higher than other comprehensive school reforms such as America ’s Choice and 
Success for AIL The most recent comprehensive school reform model developed by Modern Red 
Schoolhouse (MRSH), Systems Leadership in Middle Schools (SLMS), heavily influenced by 
Senge (1990), calls for a reorganization of a given school’s teachers and administrators onto six 
“Action Teams,” each headed by a teacher who also serves on the school’s “Guiding Coalition” 
(see Kilgore and Reynolds, 2011 for a full description of the intervention). Each action team 
develops and focuses its activities based on the Action Team topical area: family and community 
partnerships, curriculum and instruction, data analysis, technology, professional development, 
and school climate and culture. Team membership is expected to reflect the school’s faculty 
makeup with respect to departmental and grade-level representation, years of teaching 
experience, ethnicity, gender, and personality type as indicated by MBTI testing. Kilgore and 
Reynolds (2011) draw on work in organizational change (e.g., Kotter, 1996), business (e.g., 
Surowiecki, 2004), and 1950’s structural-functionalist social psychology (e.g., Levi, Torrance, 
and Pletts, 1955) to justify their model of Action Team membership. Action Teams are meant to 
replace standing and ad hoc committees and are expected to provide the organizational 
mechanism through which the school attends to all externally and internally produced directives. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The purpose of this Institute of Education Sciences Goal 1 study (R305A090481) is 1) to explore 
the relationships among the Systems Leadership for Middle Schools (SLMS) intervention and 
student achievement outcomes and 2) to explore factors and conditions that mediate the 
relationships among the SLMS intervention and student achievement outcomes. These 
explorations will result in sets of hypotheses on the relationships between the SLMS intervention 
and a) student achievement outcomes, b) the factors and conditions that mediate the relationships 
between the intervention and student achievement outcomes, and c), the SLMS intervention and 
changes in densities and extent of collegial ties. We focus on those malleable factors under the 
control of the middle school systems and likely influenced by the intervention. Key malleable 
factors include: (a) staff participation in professional development activities, (b) access to 
expertise, (c) workplace commitment, (d) instructional coherence, (e) collegial ties, (f) classroom 
teaching, (g) teacher practices, (h) distributed leadership, (i) professional community, (j) capacity 
for school improvement, and (k) the SLMS intervention itself. 

Research Questions: 

• RQ 1 . What are the state-level, school district-level, and school-level factors that 
mediate reform in treatment and control schools? ( malleable factors: a, c, g, and j ) 

• RQ2. How do the everyday behaviors of school staff change in treatment and control 
schools? ( malleable factors: b, c, d, e,f g, h, and i ) 
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• RQ 3. How does the SLMS intervention change organizational processes in treatment 
schools as compared to control schools? ( malleable factors: a, b, e, h, and k ) 

• RQ 4. To what extent does the SLMS intervention change classroom instruction in 
treatment schools as compared to control schools? ( malleable factors: d, f and k ) 

• RQ 5. To what extent does the organizational structure of SLMS intervention enable 
schools to meet the demands of the state as compared to control schools? (primary 
malleable factors: a, g, h, and k ) 

• RQ 6. To what extent does infonnation flow improve in treatment and control schools? 
(malleable factors: b, d, g, e, and k) 

• RQ7. To what extent do the densities and number of collaborative ties change in 
treatment and control schools? (malleable factors: b, d, g, e, and k ) 

Setting: 

Description of the research location. 

Six urban public middle schools in Duval County Public Schools (DCPS), Florida participated in 
the study. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 
More than 800 middle school teachers and administrators participated in the study. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and 
duration. 

The Systems Leadership for Middle Schools (SLMS) intervention was developed by Modern 
Red Schoolhouse (MRSH), an organization self-described as “a non-profit organization that 
provides customized professional development based upon solid evidence of what works in 
highly effective schools.” The SLMS program is designed to provide support for middle school 
staff to effectively devise their own solutions to contextually specific problems. There are three 
dimensions to the SLMS program that provide the basis of the support: structure, process, and 
technical knowledge (Kilgore and Reynolds, 2011). MRSH facilitators provide on-site support 
and engage in a variety of professional development activities with school principals, MRSH 
“Action Teams,” and MRSH “Guiding Coalitions.” On-site support typically consists of three- 
day visits where MRSH facilitators meet separately with each of the six Action Teams and the 
Guiding Coalition at each treatment site. MRSH facilitators meet with school principals several 
times over the three-day visits, with longer meetings typically occurring after school. 

Research Design: 

Description of the research design. 

Originally designed as a RCT, then as a quasi-experiment, participant recruitment difficulties led 
the research team, in conjunction with the sponsor, to redesign the study in a more squarely 
systematic mixed-methods direction. The six schools participating in the study have been placed 
into treatment and control groups and have been matched as pairs. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 
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Because the ultimate goal of our exploratory project is to generate a series of testable 
propositions, our mixed-method design involves descriptive statistics generated from: student 
achievement data from the Florida Department of Education (FLDOE), school culture and 
climate data from DCPS, survey data from the SLMS Network Survey administered at four time 
points as well as the SLMS Teacher and Principal Surveys administered at a time approximating 
pre-intervention conditions as well as a post intervention time point. MRSH Attendance Logs 
for SLMS professional development sessions provide measures of implementation fidelity. The 
research team also conducted more than 60 observations of classroom teaching using a protocols 
allowing for both quantitative and qualitative data collection. Additionally, project staff 
conducted nearly 80 interviews with teachers, principals, and focus groups. Finally, the research 
team observed more than 100 professional development sessions across treatment and control 
schools. Our preliminary analyses of quantitative data are exploratory and descriptive. Cross 
tabulations and correlational analyses were juxtaposed with qualitative data across metrics 
relevant to each project research question for purposes of triangulating the empirical patterns 
emerging from each longitudinal data source. More sophisticated quantitative analyses are 
necessary to address the final two research questions on networks. These analyses are currently 
underway. Network maps will graphically represent the informal structure of school actor 
collegial ties using Frank’s (Frank, 1995, 1996) KliqueFinder algorithm for identifying cohesive 
subgroups based on the pattern of ties among actors, and will embed the subgroup boundaries in 
crystallized sociograms. Ultimately, the technique reorganizes the data to generate a more 
accessible, clearer image of the social structure that is consistent with characterizations of 
systemic social structures. Ultimately, we will model the extent to which relationships between 
colleagues at each school site change. Analysis of network maps will be used to characterize 
clusters within the community, reveal individuals and organizations with high capacity for 
linking research to education as well as identify gaps in connectivity where efforts at facilitation 
are needed to improve communication. 

Findings / Results: 

Description of the main findings with specific details. 

FLDOE Data: Trends in student achievement reveal treatment schools show greater decreases in 
student achievement scores in both reading and math than control schools. However, control 
schools show a greater decrease in writing achievement scores than treatment schools. Finally, 
control schools saw greater gains in science achievement scores than treatment schools: 

• Where the mean percent of students proficient in reading in control schools dropped from 
57% to 45%, the mean percent of students proficient in reading in treatment schools 
dropped from 60% to 45%. Hence, the mean percent of students proficient in reading 
dropped at a greater rate in treatment schools than in control schools. 

• Where the mean percent of students proficient in math in control schools dropped from 
54% to 44%, the mean percent of students proficient in math in treatment schools 
dropped from 57% to 41%. Hence, the mean percent of students proficient in math 
dropped at a greater rate in treatment schools than in control schools. 

• Where the mean percent of students proficient in science in control schools increased 
from 35% to 38%, the mean percent of students proficient in science in treatment schools 
increased from 39% to 40%. 
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• Where the mean percent of student proficient in writing in control schools dropped from 
93% to 77%, the mean percent of students proficient in writing in treatment schools 
increased from 39% to 40%. 

DCPS Data: Trends in teachers’ grading of their own schools before and during the study period 
reveal a steady decrease in the percentage of teachers in treatment schools grading their school 
either “A” or “B” and a steady increase in the percentage of teachers in treatment schools 
grading their school either “D” or “F.” Control schools show an inverse pattern, revealing a 
steady increase in the percentage of teachers in control schools grading their school either “A” or 
“B” and a steady decrease in the percentage of teachers grading their school either “D” or “F.” 

While all analyses are still ongoing, tentative and preliminary findings from both deductive and 
inductive analyses of qualitative data are beginning to identify cognitive factors that may serve to 
account for the counterintuitive quantitative trends. More specifically, interview and observation 
data are beginning to show that many teachers at treatment schools interpreted the intervention as 
“one more thing,” and thusly were seemingly less motivated to complete the tasks associated 
with the intervention than the seeming minority of teachers that had “bought-in” and/or 
expressed a sense of “ownership.” 

Finally, MRSH’s SLMS Attendance Logs reveal mixed levels of fidelity to the SLMS model. 
Each of the six “Action Teams” at each treatment school site were charged with participating in 
at least 14 professional development (PD) training sessions. This was not the case for each 
Action Team at each treatment school site. Moreover, teacher-comprised Action Teams were 
charged with meeting on their own (typically bi-weekly), making meeting agendas, and keeping 
meeting minutes. Field researchers discovered that only Action Teams at one of the three 
treatment schools held regular meetings, made agendas, and kept minutes. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

We make only tentative conclusions at this time, as analyses are ongoing and the study is not 
currently scheduled to commence until June 2013. Similarly, we make only tentative 
recommendations. Tentatively, we conclude the SLMS intervention shows promise for boosting 
student achievement outcomes, despite the descriptive quantitative trends implying otherwise. 
First, we’re comfortable with this tentative conclusion at the time of writing given the 
intervention was implemented with low levels of fidelity. Second, though the case for a seeming 
minority of participants, those participants championing the intervention and carrying out 
associated tasks on a day-to-day basis steadfastly assert the intervention could “make this school 
a better place.” Finally, we are beginning to see signs suggesting the structural dimension of the 
intervention indeed provides a form of school organization ripe for fostering vast and dense 
collegial ties not immediately evident in control schools. In terms of the “process” and 
“technical knowledge” dimensions of the SLMS intervention, we tentatively conclude these 
dimensions must be redesigned and further developed by MRSH in conjunction with study 
participants and researchers. In the name of exploring how contradictory findings can be used in 
future program design and implementation, we tentatively recommend the creation of a task 
force comprised of members from key stakeholders upon completion of the study. This task 
force should be comprised of MRSH developers, study participants, and researchers and should 
include a three-person leadership team comprised of a chair from each of the three populations. 
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